Continuous motion controls operable using neurological data

ABSTRACT

Computer systems, methods, and storage media for generating a continuous motion control using neurological data and for associating the continuous motion control with a continuous user interface control to enable analog control of the user interface control. The user interface control is modulated through a user&#39;s physical movements within a continuous range of motion associated with the continuous motion control. The continuous motion control enables fine-tuned and continuous control of the corresponding user interface control as opposed to control limited to a small number of discrete settings.

BACKGROUND

Neurological data can be gathered through a variety of techniques. Onenon-invasive technique is electroencephalography (EEG), which involvesthe placement of electrodes along the scalp of a user or subject tomeasure voltage fluctuations resulting from ionic current within theneurons of the brain. EEG is often used in clinical contexts to monitorsleep patterns or to diagnose epilepsy.

Computer system user interfaces typically include a variety of userinterface controls enabling a user to interact with the computer systemthrough the user interface. In most circumstances, the user interfacecontrols rely on various input/output devices, such as keyboards,touchpads, mouse controls, game controllers, and other devices typicallyrequiring the user to use his/her hands, other body part to physicallymanipulate the hardware device.

Various “hands free” controls have been developed. However, these sufferfrom many limitations. Those that rely on voice controls typicallycannot offer the same level of precision control as through handcontrols, and are limited in the number of applications in which theyare suitable. Some may rely on camera systems to track user movement.However, reliance on a camera system inherently requires a continualview of the relevant parts of the user at a sufficient resolution, whichlimits the number of suitable applications for use.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments describedabove. Rather, this background is only provided to illustrate exemplarytechnology areas where some embodiments described herein may bepracticed.

BRIEF SUMMARY

The present disclosure relates to computer systems, methods, andcomputer storage media for using neurological data to generate acontinuous motion control (“CMC”) and associating the CMC with a userinterface control (“UIC”). The CMC is mapped to neurological datagenerated while a user performs a set of physical movements within acontinuous range of motion. The CMC is operable to modulate theassociated UIC such that neurological signals/data generated during auser's physical movements within the continuous range of motion serve asinput for controlling the UIC in a continuous/analog fashion.

At least some of the embodiments described herein provide fine, analogcontrol of one or more user interface operations having continuoussettings as opposed to a limited number of discrete settings. In someembodiments, a CMC is generated through machine learning and/orregression techniques so that neurological data are converted to scalarnumbers as opposed to simply being classified into one of a limitednumber of discrete categories.

In some embodiments, a CMC operates through neurological data generatedduring physical movement of one or more of a hand, foot, face, arm, leg,head, and/or other body part. Exemplary continuous motions that may beassociated with a CMC include foot flexion and extension movements, handrotation movements, facial movements (e.g., smiling, brow furrowing,opening of eyes to various degrees, mouth opening and closingmovements), arm raising and lowering movements, and other physicalmovements that are part of a continuous range of motion capable of beingperformed by a user.

In some embodiments, the UIC associated with the CMC is one or more of adisplay control, audio control (e.g., volume control), navigationcontrol, system setting control, control related to an avatar or one ormore other character (e.g., facial expressions or other charactermovements that track a user's movements), gaming controls, menu control,or other user interface operation or control.

In some embodiments, a generic CMC is constructed by obtainingneurological data generated by a plurality of users while the usersperform various physical movements within a particular range of motionand mapping the obtained neurological data to those physical movements.The resulting generic CMC is then fine-tuned according to a particularuser's unique neurological profile to thereby calibrate the CMC as anindividualized CMC. Accordingly, the generic CMC is operable as abaseline for efficiently upgrading to an individualized CMC.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates a computing environment that can be used to generatea continuous motion control for analog control of a user interfacecontrol;

FIG. 2 illustrates a flowchart of an exemplary method for usingneurological data to modulate a continuous user interface control;

FIG. 3 illustrates EEG data representing a generic continuous motioncontrol and individual EEG data generated by a particular user for usein a continuous motion control calibration process;

FIGS. 4A-4C illustrate operation of an exemplary continuous motioncontrol configured to function according to motion of a user's hand tomodulate a volume control;

FIGS. 5A-5C illustrate operation of an exemplary continuous motioncontrol configured to function according to motion of a user's foot tomodulate a virtual pedal;

FIGS. 6A-6C illustrate operation of an exemplary continuous motioncontrol configured to function according to motion of a user's thumb tomodulate a virtual joystick; and

FIGS. 7A-7C illustrate operation of an exemplary continuous motioncontrol configured to function according to a hand swiping motion tomodulate a carousel menu.

DETAILED DESCRIPTION

The present disclosure relates to computer systems, methods, andcomputer storage media for generating a continuous motion control(“CMC”) using neurological data and for associating the CMC with a userinterface control (“UIC”) to enable analog control of the UIC throughphysical movements within a continuous range of motion. Neurologicaldata of a user generated during physical movements within the continuousrange of motion are mapped to the physical movements in order togenerate the CMC. The CMC is associated with a UIC such thatneurological signals generated during a user's physical movements withinthe continuous range of motion serve as input for controlling the UIC ina continuous/analog fashion.

Various technical effects and benefits may be achieved by implementingaspects of the disclosed embodiments. By way of example, the disclosedembodiments are operable to enable the use of neurological signals forfine control of one or more UICs without the need of a keyboard,handheld controller, mouse, joystick, and/or other traditional inputhardware.

Further, by configuring the CMCs to operate using neurological dataassociated with a user's physical movements, the need for a camera todirectly track the physical movements is reduced or eliminated. Thisprovides hands-free controller functionality to technologicalenvironments in which such functionality was not previously feasible.For example, in various virtual reality or augmented realitytechnological environments, which typically rely on some form ofheadgear as the basic hardware platform, it is not always practical orpossible to position a camera on or within the headgear in a manner thatcan sufficiently capture a user's physical movements, particularly anymovements beyond a limited portion of the face, such as hand and footmovements.

Technical effects and benefits further include enabling control ofcontinuous-type UICs through neurological signals, as opposed to controlbeing limited to operations that rely on discrete categorydeterminations (e.g., a binary on/off control). By way of example, a CMCas disclosed herein provides analog control functionality, and allowsfor fine, analog control of operations having continuous levels/settingsas opposed to a limited number of discrete settings.

In addition, a CMC as disclosed herein is configured to provide gradientinput to enable analog control of an associated user interface control.For example, in some embodiments the CMC is generated using machinelearning and/or regression techniques so that neurological signals areconverted to scalar numbers as opposed to simply being classified intoone of a limited number of discrete classes. In this manner, thegranularity of control over an associated UIC is limited only byhardware limitations and the preset or inherent granularity of the UICitself, and not by the CMC. This is particularly true when the CMC hasbeen tuned or otherwise customized for a particular user or set ofusers.

The term “continuous,” as used herein with respect to “continuous motioncontrol,” “continuous user interface control,” and similar terms, refersto a control that is not restricted in granularity to mere binary valuesor a limited set of values that are more restrictive than therestrictions that are imposed by hardware limitations, number roundinglimits (e.g., rounding limits built into one or more models underlying aCMC), or other limits external to the control itself. Terms such as“analog control” therefore refer to controls having improved orincreased granularity when compared to binary controls or limited setcontrols. Although a large number of sequentially arranged discretecategories may, in some circumstances, give the impression of a somewhatcontinuous control, preferred embodiments of CMCs are configured toprovide a continuous gradient of at least 5 or more, at least 10 ormore, at least 20 or more, at least 50 or more, or at least 100 or moremeasurably different levels, settings, or responses when applied tooperate a UIC.

In some embodiments, a CMC involves movement of a hand, foot, face, arm,leg, head, and/or other body part. Examples of continuous motionssuitable for application as continuous motion controllers include footflexion and extension movements, hand rotation movements, facialmovements (e.g., smiling, brow furrowing, opening of eyes to variousdegrees, mouth opening and closing movements), arm raising and loweringmovements, etc.

UICs that may be associated with a CMC include display controls, audiocontrols (e.g., volume control), navigation controls, system settingcontrols, controls related to avatars or other characters (e.g., facialexpressions, like the movement of a mouth or eye, or any other charactermovements that track a user's movements), gaming controls, and the like.Although not limited to such, CMCs disclosed herein are particularlyuseful for controlling continuous UICs that have a continuous gradientrange of levels, settings, positions, or other responses, as opposed toa limited number of discrete settings.

Neurological signals used to gather neurological data for generating acontinuous motion control and/or for powering a CMC during use may begathered using EEG. Other embodiments may utilize neurological datagathered through other means, in addition to or alternative to EEG, suchas magnetoencephalography (MEG), functional magnetic resonance imaging(fMRI), or other techniques for gathering context-based neurologicaldata. In presently preferred embodiments and recited claims,non-invasive EEG techniques are used. It will be appreciated, however,that the scope of this application also covers embodiments in which thedescribed/claimed EEG is replaced and/or supplemented with the MEG, fMRIand/or other context-based neurological data.

In this description and in the claims, the term “computing system” or“computer architecture” is defined broadly as including any standaloneor distributed device(s) and/or system(s) that include at least onephysical and tangible processor, and a physical and tangible memorycapable of having thereon computer-executable instructions that may beexecuted by the processor(s).

FIG. 1 illustrates an exemplary computer system 100 in which a CMC isgenerated from neurological data and is associated with a UIC to enableanalog control over the associated UIC through neurological input. Theillustrated embodiment includes a CMC generator 120 in communicationwith a local system 130. The CMC generator 120 and the local system 130are connected by (or are part of) a network 110, such as, for example, aLocal Area Network (“LAN”), a Wide Area Network (“WAN”), and even theInternet.

As shown, the illustrated CMC generator 120 includes memory 122, and theillustrated local system 130 includes memory 132. Each of the CMCgenerator 120 and the local system 130 include at least one processor124 and 134, respectively. The memory 122 and 132 may independently bephysical system memory, which may be volatile, non-volatile, or somecombination of the two. The term “memory” may also be used herein torefer to non-volatile mass storage such as physical storage media.

The illustrated system 100 includes an EEG sensor 140 through which auser provides neurological input to the local system 130. The CMCgenerator 120 and the local system 130 can include other input/outputhardware 126 and 136, including one or more keyboards, mouse controls,touch screens, microphones, speakers, display screens, track balls, andthe like to enable the receiving of information from a user and fordisplaying or otherwise communicating information to a user.

The CMC generator 120 and the local system 130 include a number ofexecutable modules or executable components 128 a-128 b and 138 a-138 c.As used herein, the term “executable module” or “executable component”can refer to software objects, routings, or methods that may be executedon the computing system. The different components, modules, engines, andservices described herein may be implemented as objects or processesthat execute on the computing system (e.g., as separate threads).

The various components illustrated in FIG. 1 represent only a fewexample implementations of a computer system for generating a CMC foranalog control of a UIC. Other embodiments may divide the describedmemory/storage data, modules, components, and/or functions differentlyamong the CMC generator 120 and the local system 130, and someembodiments may move more of the processing toward the local system 130than the CMC generator 120, or vice versa, relative to the particularembodiment illustrated in FIG. 1. In some embodiments, memory componentsand/or program modules are distributed across a plurality of constituentcomputer systems in a distributed environment. In other embodiments,memory components and program modules are included in a singleintegrated computer system. Accordingly, the systems and methodsdescribed herein are not intended to be limited based on the particularlocation at which the described components are located and/or at whichtheir functions are performed.

According to the illustrated embodiment, the memory 122 is used forstoring crowd-based neurological data 122 a, which includes neurologicaldata (e.g., EEG signal data) gathered from a plurality of subjects. Atleast some of the crowd-based neurological data 122 a relates tocontext-specific neurological data, such as particular EEG signalsgenerated during different physical movements within a particularcontinuous range of motion. For example, a particular continuous rangeof motion may be flexion/extension of a foot, and a physical movementwithin the flexion/extension range may be positioning the foot at aparticular angle relative to the leg (e.g., 90-160 degrees of extensionfrom the leg). Other physical movements within the flexion/extensionrange may include other degrees of foot extension (besides 90-160degrees of extension), as well as other types of positions andmovements, including lateral movements, circular rotations, heal raises,toe movements and/or any other type of foot movement(s). Accordingly,the crowd-based neurological data 122 a is correlated to a continuousrange of motion library 122 b and a physical movement library 122 c fora variety of different physical movements corresponding to footmovements, as well as other body movement.

Other examples of continuous ranges of movement include degree of mouthopening, degree of eye opening, head tilting and/or turning, therotating of a thumb about an arc (e.g., as in a “thumb joystick”), handrotation, degree of wrist extension, degree of arm or leg raising,degree of squatting, degree of torso twisting, and/or any othercontinuous range of motion capable of being performed by a user.

In some embodiments, the continuous range of movement also includes acontinuous range of force being applied, even when movement is noteasily detected, is non-existent, or is not commensurate with the forcethat is applied. For instance, the range in forces applied when closingeyes, clenching a fist, pinching or flexing muscles are also capable ofbeing mapped with EEG and other neurological data. In some instances,the scope of the disclosed embodiments for detecting a ‘continuousmovement’ and/or ‘continuous range of motion’ includes the detection ofthe continuous gradient of forces that are applied by one or more bodyparts.

Additionally, the different physical movements within a given continuousrange of motion may be measured/detected differently based on position(as in the example of different degrees of foot extension) or based onsome other variable, such as the velocity or speed of the movement. Byway of example, for a continuous range of motion defined as ahand-swiping motion, different physical movements within the range ofmotion may include the angular degree of the hand swipe (i.e., aposition-based variable) and/or may include the velocity of the hand orfinger(s) during the swiping action (i.e., a speed-based variable). Suchembodiments are particularly useful for controlling UICs that operatealong similar variables. For example, a menu of options presented in acarousel format can be controlled through a hand swiping motion, withthe carousel configured to spin with a speed proportional to the speedof the hand-swipe.

Preferably, for each continuous range of motion, the relatedneurological data includes neurological signals associated with enoughdifferent physical movements within the continuous range of motion,collected from enough different subjects, to generate a generic CMC 122d for the continuous range of motion, as explained in more detail below.

The illustrated CMC generator 120 includes a model generator 128 aconfigured to receive the crowd-based neurological data 122 a, and for aparticular continuous range of motion, to generate a model thatmaps/relates the neurological data to the physical movements within thecontinuous range of motion. In some embodiments, the model generator 128a includes signal processing functionality to provide filtering (e.g.,low and/or high band-pass filtering and/or filtering of delta and gammaEEG waves), artifact removal (e.g., removal of common EEG artifactsknown to be associated with blinks, yawns, audio or visual stimulation,or other data and movements that are not correlated to the targetedmovement of a particular application), and/or other processing of one ormore signals of the crowd-based neurological data 122 a.

In some embodiments, the model generator 128 a is operable to performregression analysis on the crowd-based neurological data 122 a todetermine the corresponding generic CMCs 122 d. For example, differentphysical movements within a particular continuous range of motion may becorrelated to different percentages of the power spectrum (e.g., asdetermined through Fourier analysis) within the different wave bands(alpha, beta, gamma, delta) of the corresponding EEG signals, may becorrelated to an amount of phase and/or magnitude synchrony, and/or maybe correlated to other characteristics of the corresponding EEG signals.

In preferred embodiments, the model generator 128 a is configured to usemachine learning techniques to correlate EEG signal information tocorresponding physical movements within a particular continuous range ofmotion in order to generate a predictive model that is operable togenerate signals that reflect or associate physical movements (position,speed, force, etc.), as output, based on detected neurological signalinput.

As shown, the CMC generator 120 also includes a communication module 128b configured to enable the CMC generator 120 to communicate with one ormore other systems, including the local system 130 (e.g., in conjunctionwith communication module 138 b). In some embodiments, the communicationmodules 128 b and 138 b provide application programming interfaces(APIs) that enable the respective systems to communicate and share datawith one another and/or with other systems (including other systemswithin a distributed computing environment).

In the illustrated embodiment, the local system 130 includes memory 132.Memory 132 includes individualized neurological data 132 a, whichincludes neurological data (e.g., EEG signal data) associated with anidentified individual. For example, the individualized neurological data132 a may be associated with user profiles 132 e. As with thecrowd-based neurological data 122 a, the individualized neurologicaldata 132 a includes neurological data from different physical movements(e.g., stored in physical movements library 132 c) within a particularcontinuous range of motion (e.g., as stored in continuous range ofmotion library 132 b). However, each piece of individualizedneurological data 132 a is associated with a specific individual suchthat, for a particular continuous range of motion, a set of anindividual's EEG signatures are associated with a corresponding set ofthe individual's physical movements within the continuous range ofmotion.

The illustrated local system 130 also includes a calibration module 138a configured to tune or otherwise modify a generic CMC to a specificindividual or set of individuals, according to the individualizedneurological data 132 a corresponding to the individual(s). For example,for a particular continuous range of motion, the calibration module 138a operates to associate the individual's corresponding neurological data132 a to the corresponding generic CMC received from the CMC generator120. The calibration module 138 a operates to adjust/customize thegeneric CMC to the specific individual(s). In some instances, thecalibrated CMCs are stored as individualized CMCs 132 d at the localsystem 130. Although, they may also be stored at a location remote tothe local system 130. For instance, the individualized CMCs 132 d may bestored at a third party game server that accesses and applied theindividualized CMCs 132 d, when necessary.

The calibration module 138 a can provide a number of advantages. Becausea specific individual will typically not produce exactly the same EEGsignature as another individual, even for the same physical movements,use of the calibration module 138 a can better tailor the CMC to thespecific individual, providing better and more accurate responsivenesswhen the individualized CMC is associated with a UIC and put into use.In addition, despite the differences, EEG signatures for a particularphysical movement will have many similarities across all or most users.Certain embodiments disclosed herein therefore provide a generic CMCgenerated from the neurological data of a plurality of users, thegeneric CMC establishing a baseline from which an individual CMC can betailored and fine-tuned, if desired.

When the CMC is tuned for a limited group of users (e.g., a family, ateam, a company, etc.), based on trained data for the group, thecustomized CMC will still provide better accuracy and effective controlthan the generic CMC that is based on all users.

Another example of a benefit of this configuration is that it provides auser or defined group with the option for immediate use of a generic CMCor the option to set up a customized CMC. For example, a guest user mayonly want to use an application for a short time, and may not want tofirst set up a personal profile or run through an initiation sequence toestablish a CMC. In many instances, the generic CMC will providesufficient responsiveness and accuracy to allow the user to beginimmediate use of the application and control of one or more UICs withinthe application with little to no initial setup of the CMC. At the sametime, other users may wish to establish more customized and/or bettertailored CMCs that improve upon the default generic CMC. These users mayutilize the calibration module 138 a to generate their respectiveindividualized CMCs 132 d.

As shown, the local system 130 also includes a user interfaceapplication (“UI application”) 150 operable on the local system 130. TheUI application 130 may be a video game, a virtual reality or augmentedreality simulator, an audio or audiovisual service, a word processor, aspreadsheet application, a database manager, or any other applicationcapable of receiving input through one or more UICs. The illustrated UIapplication 150 includes a number of UICs 152 a to 152 n (referred togenerically as 152). As described herein, the UICs 152 may be displaycontrols, audio controls, character movement or character actioncontrols, menu controls, navigation controls, or other controls by whicha user interacts with the application 150 to modulate input or output,settings, parameter levels, etc.

The local system 130 includes an associating module 138 c configured toassociate one or more CMCs (either generic CMCs 122 d or individualizedCMCs 132 d) with one or more corresponding UICs 152 of the application150. In some embodiments, the association may be configured according touser preferences and selections. For example, a user may wish toassociate a particular range of motion with a particular UIC 152, andmay wish to define the relationship between the physical movements ofthe range of motion and the response of the UIC 152. In otherembodiments, one or more CMCs are automatically assigned to particularUICs 152 (e.g., by default). For example, the particular application 150running on the local system 130 may define the relationship between theone or more CMCs and the one or more UICs. By way of example, anapplication 150 may define what range of motion, and therefore what CMC,controls a particular UIC, and/or may define what physical movementswithin the range of motion correspond to responses of the particular UICaccording to pre-set or default application settings.

In the description that follows, embodiments are described withreference to acts that are performed by one or more computing systems.If such acts are implemented in software, one or more processors of theassociated computing system that performs the act direct the operationof the computing system in response to having executedcomputer-executable instructions. For example, such computer-executableinstructions may be embodied on one or more computer-readable media thatform a computer program product. An example of such an operationinvolves the manipulation of data. The computer-executable instructions(and the manipulated data) may be stored in the memory 122 of the CMCgenerator 120, the memory 132 of a local system 130, and/or in one ormore separate computer system components (e.g., in a distributedcomputer system environment).

The computer-executable instructions may be used to implement and/orinstantiate all of the functionality disclosed herein, including thefunctionality that is disclosed in reference to the flow diagram of FIG.2.

FIG. 2 is a flowchart 200 of a method for using neurological data tomodulate a continuous user interface control. A computer system createsa CMC that maps neurological data obtained from a plurality of users toa set of physical movements within a continuous range of motion of theplurality of users (act 210). The mapping may be a mapping that usesaveraging, best fit algorithms or any other statistical or correlationmapping algorithms.

The act of creating the CMC (act 210) optionally includes obtaining theneurological data from a plurality of users and building a database ofobtained neurological data from a plurality of users. This may beaccomplished, for example, by measuring EEG data for a plurality ofusers, as described above, while the users perform certain tasks.Alternatively, or additionally, act 210 further includes accessing thestored neurological data from a database that already contains thestored data from a plurality of users. The CMC data may be mapped to allusers, to discrete groups of users and to individual users.

As described herein, the continuous range of motion may includemovements of a hand, foot, arm, leg, face, head, or other body part, andthe different physical movements within the continuous range of movementmay be differentiated based on position within the range of motion(e.g., rotation angle of hand, height arm is raised), direction ofmovement (e.g., clockwise rotation vs. counterclockwise rotation,flexion vs. extension, etc.), force(s) exerted (with or without a changein relative position), and/or speed of movement through all or part ofthe range of motion (e.g., speed at which hand is swiped).

The computer system then tunes/calibrates the CMC to a particular user(or group) by at least mapping neurological data obtained from theparticular user (or group) while the particular user is performing theset of physical movements within the continuous range of motion (act220). When the CMC is tuned to a group, the group averages may be usedto define the limits and rates of change for applying the CMC during useby the group.

It will be appreciated that the creation of a first baseline/genericCMC, which is subsequently tuned for specific individual(s), isparticularly beneficial for enabling broad application and scalabilityto the disclosed applications of use. For instance, in at least someembodiments, this process saves overall time and processing requirementsrequired to build a plurality of CMCs that are customized to particularindividuals and/or groups, rather than requiring their construction fromscratch.

The computer system then associates the CMC to a continuous UIC (act230). As described by the foregoing, the UIC may be a display control,audio control, character movement or character action control, menucontrol, navigation control, or other control by which a user interactswith a user interface to modulate input or output, settings, parameterlevels, and the like. The embodiments described herein are particularlyuseful with continuous UICs that operate with a continuous range ofsettings/responses as opposed to a limited number of discretesettings/responses.

In the illustrated embodiment, the computer system subsequently detectsa user input comprising neurological data associated with a physicalmovement within the continuous range of movement (act 240). In presentlypreferred embodiments, the neurological data is received by the computersystem from an EEG sensor or other sensor worn by the user. In someembodiments, this data is obtained as the user performs the physicalmovement. In other embodiments, the data is obtained as the user simplyfocuses on performing the physical movement, similar to how an amputeemight think about moving an amputated limb.

The computer system uses the generated CMC to modulate the continuousUIC in a manner that correlates with the physical movement of the userwithin the continuous range of motion (act 250). For example, theposition, direction, force and/or speed of the particular physicalmovement can define the response of the continuous UIC. FIG. 3illustrates an exemplary calibration step for generating anindividualized CMC based on a generic CMC modified according to theindividual's neurological data. FIG. 3 illustrates a chart 300 showing,for a particular continuous range of motion, the relationship betweenEEG machine learning output (axis 310) and the predicted relative degreeof physical movement within the continuous range of motion (axis 320).As described by the foregoing, some embodiments utilize machine learningtechniques for discovering correlations and relationships between one ormore physical movements and the EEG data generated during thosemovements. The illustrated EEG machine learning output thereforerepresents one or more EEG data characteristics determined to becorrelated with physical movements within the continuous range ofmotion.

In this particular example, the EEG machine learning output is relatedto the degree of extension of a foot. The chart 300 illustrates arepresentation of a generic CMC (series 330) showing a determinedrelationship between EEG output and foot extension based on the EEG dataof a plurality of subjects. The chart 300 also illustrates a pluralityof data points (series 340) representing a particular individual's EEGoutput (according to the determined machine learning variables) andcorresponding degree of foot extension associated with those EEGoutputs.

The series 340 is generated during a calibration process. For example,an application can guide the user to extend his/her foot at a number ofdifferent degrees of extension while the corresponding EEG data aregathered. The number of data points collected may vary. In someembodiments, the number, concentration, and/or granularity of datapoints collected depends on the particular range of motion involved,user preferences, level of fine-tuning desired from the calibrationprocess, sensitivity of hardware, and/or timing preferences, forexample.

As shown, the resulting individual series 340 is not completely alignedwith the generic series 330. An individualized CMC is generated byadjusting the corresponding generic CMC by the calibration offset(represented by spaces 350) to arrive at the individualized CMC.

The following examples represent operation and functionality of variouscontinuous UICs associated with continuous CMCs. The examples areillustrative only. The illustrated UICs may be associated with otherCMCs (i.e., other arrangements of ranges of motion and/or physicalmovements within the range of motion), and vice versa. In addition,other embodiments include other CMCs based on other physical movementsand ranges of motion and/or other UICs.

FIGS. 4A-4C illustrate operation of a CMC configured to functionaccording to rotation of a user's hand 410 in order to modulate a volumecontrol 420. As shown in FIG. 4A, the CMC is configured so that as theuser's hand 410 is in a first position, with the knuckles up at 0degrees, the corresponding volume control 420 is set at zero. FIG. 4Billustrates that as the user rotates his/her hand 410 to a secondposition, the corresponding neurological signature generated by the usercauses the volume control 420 to respond with a corresponding increasein volume level. FIG. 4C illustrates that as the user rotates his/herhand 410 even further to a third position, the correspondingneurological signature generated by the user causes the volume control420 to further respond with a corresponding further increase in volumelevel. In some embodiments, rotation in the opposite direction operatesto decrease the volume level. Although FIGS. 4A-4C show distinct volumelevels at distinct hand rotation positions, it should be understood thata continuous range of volume levels between each illustrated positionare possible, according to relative hand rotation positions between theillustrated hand rotation positions.

FIGS. 5A-5C illustrate operation of a CMC configured to functionaccording to degree of extension of a user's foot 510 in order tomodulate depression of a virtual gas pedal 520 in a video gameapplication. As shown in FIG. 5A, the CMC is configured so that as theuser's foot 510 is positioned at full flexion, the corresponding virtualgas pedal 520 is positioned in an upright (and not depressed) position.FIG. 5B illustrates that as the user extends his/her foot 510 to a moreextended position, the corresponding neurological signature generated bythe user causes the virtual gas pedal 520 to respond with acorresponding degree of depression. FIG. 5C illustrates that as the userextends his/her foot 510 even further, the corresponding neurologicalsignature generated by the user causes the virtual gas pedal 520 tofurther respond with a corresponding further degree of depression.Although FIGS. 5A-5C show distinct gas pedal positions in relation todistinct foot extension positions, it should be understood that acontinuous range of gas pedal positions are possible in between theillustrated positions, corresponding to the continuous range of footextension of the user.

FIGS. 6A-6C illustrate operation of a CMC configured to functionaccording to movements of a user's thumb 610 (e.g., combinations offlexion, extension, abduction, adduction, and opposition movements) inorder to modulate actuation of a virtual joystick 620. As shown in FIG.6A, the CMC is configured so that as the user's thumb 610 is flexed andslightly adducted (the thumb of a left hand, in this example), thecorresponding neurological signature generated by the user causes thevirtual joystick 620 to be positioned a corresponding degree upwards andslightly to the left. FIG. 6B illustrates that as the user furtheradducts his/her thumb 610, the corresponding neurological signaturegenerated by the user causes the virtual joystick 620 to be positioned acorresponding degree further to the left. FIG. 6C illustrates that asthe user shifts his/her thumb to an abducted and more extended position,the corresponding neurological signature generated by the user causesthe virtual joystick 620 to move to a downward/right diagonal position.Although FIGS. 6A-6C show distinct joystick positions in relation todistinct thumb positions, it should be understood that other positionscorresponding to other thumb positions/movements are possible, and thata continuous range of joystick positions are possible in between theillustrated positions or other positions, corresponding to thecontinuous ranges of thumb movements of the user.

FIGS. 7A-7C illustrate operation of a CMC configured to functionaccording to the swipe speed of a user's hand 710 in order to modulaterotation of a carousel menu 720. As shown in FIG. 7A, the CMC isconfigured so that as the user's hand 710 is held still, thecorresponding carousel menu is held still. FIG. 7B illustrates that asthe user swipes his/her hand 710 in a relatively slow motion, thecorresponding neurological signature generated by the user causes thecarousel menu 720 to respond with a correspondingly slow rotation. FIG.7C illustrates that as the user swipes his/her hand 710 in a relativelyfast manner, the corresponding neurological signature generated by theuser causes the carousel menu 720 to respond with a correspondingly fastrotation. Other rotation speeds are also possible at variousintermediate degrees, in a continuous fashion, according to relativehand swipe speed.

While specific examples of use have been provided, it will beappreciated that the scope of this disclosure also applies to otheruses, including uses in which

EEG based on detected force/pressure is used to modulate the CMC.Likewise, applications of use include other UI controls, such ascontrols for changing brightness of a display or lighting or filterlevels that are applied. The applications of use and CMCs that arecontrollable through application of this disclosure also includecontrols for mechanical tools and machinery, such as controls for movingand operating robotic arms and other tools.

Embodiments of the present invention may comprise or utilize aspecial-purpose or general-purpose computer system that includescomputer hardware, such as, for example, one or more processors andsystem memory. Embodiments within the scope of the present inventionalso include physical and other computer-readable media for carrying orstoring computer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general-purpose or special-purpose computer system.Computer-readable media that store computer-executable instructionsand/or data structures are computer storage media. Computer-readablemedia that carry computer-executable instructions and/or data structuresare transmission media. Thus, by way of example, and not limitation,embodiments of the invention can comprise at least two distinctlydifferent kinds of computer-readable media: computer storage media andtransmission media.

Computer storage media are physical storage media that storecomputer-executable instructions and/or data structures. Physicalstorage media include computer hardware, such as RAM, ROM, EEPROM, solidstate drives (“SSDs”), flash memory, phase-change memory (“PCM”),optical disk storage, magnetic disk storage or other magnetic storagedevices, or any other hardware storage device(s) which can be used tostore program code in the form of computer-executable instructions ordata structures, which can be accessed and executed by a general-purposeor special-purpose computer system to implement the disclosedfunctionality of the invention.

Transmission media can include a network and/or data links which can beused to carry program code in the form of computer-executableinstructions or data structures, and which can be accessed by ageneral-purpose or special-purpose computer system. A “network” isdefined as one or more data links that enable the transport ofelectronic data between computer systems and/or modules and/or otherelectronic devices. When information is transferred or provided over anetwork or another communications connection (either hardwired,wireless, or a combination of hardwired or wireless) to a computersystem, the computer system may view the connection as transmissionmedia. Combinations of the above should also be included within thescope of computer-readable media.

Further, upon reaching various computer system components, program codein the form of computer-executable instructions or data structures canbe transferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a “NIC”), and theneventually transferred to computer system RAM and/or to less volatilecomputer storage media at a computer system. Thus, it should beunderstood that computer storage media can be included in computersystem components that also (or even primarily) utilize transmissionmedia.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at one or more processors, cause ageneral-purpose computer system, special-purpose computer system, orspecial-purpose processing device to perform a certain function or groupof functions. Computer-executable instructions may be, for example,binaries, intermediate format instructions such as assembly language, oreven source code.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, virtual or augmented realityheadsets, and the like. The invention may also be practiced indistributed system environments where local and remote computer systems,which are linked (either by hardwired data links, wireless data links,or by a combination of hardwired and wireless data links) through anetwork, both perform tasks. As such, in a distributed systemenvironment, a computer system may include a plurality of constituentcomputer systems. In a distributed system environment, program modulesmay be located in both local and remote memory storage devices.

Those skilled in the art will also appreciate that the invention may bepracticed in a cloud computing environment. Cloud computing environmentsmay be distributed, although this is not required. When distributed,cloud computing environments may be distributed internationally withinan organization and/or have components possessed across multipleorganizations. In this description and the following claims, “cloudcomputing” is defined as a model for enabling on-demand network accessto a shared pool of configurable computing resources (e.g., networks,servers, storage, applications, and services). The definition of “cloudcomputing” is not limited to any of the other numerous advantages thatcan be obtained from such a model when properly deployed.

A cloud computing model can be composed of various characteristics, suchas on-demand self-service, broad network access, resource pooling, rapidelasticity, measured service, and so forth. A cloud computing model mayalso come in the form of various service models such as, for example,Software as a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”). The cloud computing model may alsobe deployed using different deployment models such as private cloud,community cloud, public cloud, hybrid cloud, and so forth.

Some embodiments, such as a cloud computing environment, may comprise asystem that includes one or more hosts that are each capable of runningone or more virtual machines. During operation, virtual machines emulatean operational computing system, supporting an operating system andperhaps one or more other applications as well. In some embodiments,each host includes a hypervisor that emulates virtual resources for thevirtual machines using physical resources that are abstracted from viewof the virtual machines. The hypervisor also provides proper isolationbetween the virtual machines. Thus, from the perspective of any givenvirtual machine, the hypervisor provides the illusion that the virtualmachine is interfacing with a physical resource, even though the virtualmachine only interfaces with the appearance (e.g., a virtual resource)of a physical resource. Examples of physical resources includingprocessing capacity, memory, disk space, network bandwidth, mediadrives, and so forth.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A computer system configured for usingneurological data to modulate a continuous user interface control, thecomputer system comprising: one or more processors; and one or morehardware storage devices having stored thereon computer-executableinstructions which are executable by the one or more processors to causethe computing system to perform at least the following: create acontinuous motion control that maps neurological data obtained from aplurality of users to a set of physical movements within a continuousrange of motion of the plurality of users; tune the continuous motioncontrol to a particular user by at least mapping neurological dataobtained from the particular user while the particular user isperforming the set of physical movements within the continuous range ofmotion; associate the continuous motion control to a continuous userinterface control; detect user input comprising neurological dataassociated with a physical movement within the continuous range ofmovement; and modulate the continuous user interface control in a mannercorresponding to the physical movement of the user within the continuousrange of motion.
 2. The computing system of claim 1, wherein the set ofphysical movements within the continuous range of motion includesphysical movements that are differentiated from one another based onrelative position of a body part within the continuous range of motion.3. The computing system of claim 1, wherein the set of physicalmovements within the continuous range of motion includes physicalmovements that are differentiated from one another based on relativespeed at which the physical movements are made within the continuousrange of motion.
 4. The computing system of claim 1, wherein thecontinuous range of motion includes movements of a hand, foot, arm, leg,face, or head.
 5. The computing system of claim 1, wherein the set ofphysical movements within the continuous range of motion includesphysical movements that are differentiated from one another based onrelative force.
 6. The computing system of claim 1, wherein thecontinuous user interface control is an audio volume level control. 7.The computing system of claim 1, wherein continuous user interfacecontrol is operable in a virtual reality or an augmented realityenvironment.
 8. The computing system of claim 1, wherein the continuoususer interface control is operable with a continuous range of settingsas opposed to a limited number of discrete settings.
 9. The computingsystem of claim 1, wherein the user input comprising neurological datais obtained from an electroencephalography sensor contemporaneously wornby the user.
 10. The computing system of claim 1, wherein the continuousmotion control is operable to modulate the continuous user interfacecontrol without input obtained from a camera.
 11. The computing systemof claim 1, wherein the continuous user interface control is a virtualanalog joystick.
 12. The computing system of claim 11, wherein thecontinuous motion control is operable to modulate the virtual analogjoystick through a set of thumb movements.
 13. The computing system ofclaim 1, wherein the continuous motion control is operable to modulatethe continuous user interface control through foot movements.
 14. Thecomputing system of claim 1, wherein the continuous motion control isoperable to modulate the continuous user interface control throughfacial movements.
 15. The computing system of claim 1, wherein thecontinuous motion control maps neurological data obtained from aplurality of users using machine learning techniques.
 16. Acomputer-implemented method for using neurological data to modulate acontinuous user interface control, the method being implemented by acomputing system that includes at least one processor and one or morehardware storage devices having stored thereon computer-executableinstructions that are executable by the at least one processor to causethe computing system to implement the method, the method comprising:creating a continuous motion control that maps neurological dataobtained from a plurality of users to a set of physical movements withina continuous range of motion of the plurality of users; tuning thecontinuous motion control to a particular user by at least mappingneurological data obtained from the particular user while the particularuser is performing the set of physical movements within the continuousrange of motion; associating the continuous motion control to acontinuous user interface control; detecting user input comprisingneurological data associated with a physical movement within thecontinuous range of movement; and modulating the continuous userinterface control in a manner corresponding to the physical movement ofthe user within the continuous range of motion.
 17. The method of claim16, wherein the set of physical movements within the continuous range ofmotion includes physical movements that are differentiated from oneanother based on relative position of a body part within the continuousrange of motion and based on relative speed at which the physicalmovements are made within the continuous range of motion.
 18. The methodof claim 16, wherein the continuous motion control is operable tomodulate the continuous user interface control in an analog fashion. 19.The method of claim 16, wherein the continuous motion control isoperable to modulate the continuous user interface control without inputobtained from a camera.
 20. One or more hardware storage device havingstored thereon computer-executable instructions which are executable byone or more processors of a computing system to use neurological data tomodulate a continuous user interface control by at least causing thecomputer system to perform a method that includes the following:creating a continuous motion control that maps neurological dataobtained from a plurality of users to a set of physical movements withina continuous range of motion of the plurality of users; tuning thecontinuous motion control to a particular user by at least mappingneurological data obtained from the particular user while the particularuser is performing the set of physical movements within the continuousrange of motion; associating the continuous motion control to acontinuous user interface control; detecting user input comprisingneurological data associated with a physical movement within thecontinuous range of movement; and modulating the continuous userinterface control in a manner corresponding to the physical movement ofthe user within the continuous range of motion.